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Abstract: The purpose of the present study was twofold, namely implementing a multifunctional (self-) 
evaluation instrument for physical education in a sample of Flemish secondary schools (N=100), while 
simultaneously obtaining a preliminary picture of the subjects’ product and process quality. Descriptive statistics 
revealed that P. E. teachers’ engagement with regard to the provision of quality school physical education is in 
essence restricted to initiatives and agreements that directly influence the personal class room setting, while 
general quality management principles and/or initiatives (e.g., strategic planning), aimed at assuring and/or 
improving the subjects’ overall professionalism, status and/or situation, are at present in many schools 
underdeveloped. From the perspective of introducing general quality management principles in school physical 
education, a large majority of participating P. E. teachers exhibit great interest in and appreciation for the 
IKLO-instrument. When applied correctly, this instrument is considered to be a valuable asset for the future of 
school physical education. 
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1. Introduction 

1.1 The situational context of school physical education 

Over the past decade, the interest in the quality of school physical education has increased, following an 
extensive amount of scientific research on the possible benefits (Dale & Corbin, 2000; Emmanouel, Zervas & 
Vagenas, 1992; Le Masurier & Corbin, 2006) and the discrepant day-to-day reality of the subject (Hardman & 
Marshall, 2000; Laws, 2002, 2003; Piihse & Gerber, 2005). On a political level, this resulted in a plethora of 
national and international declarations, recommendations and guidelines aimed at improving the access to, and the 
provision of, quality school physical education (CDC, 1997; EUPEA, 2002; NASPE, 2004). Next to this, different 
researchers have started to investigate the quality of school physical education (Brettschneider, et al., 2006; Egger, 
2001; Stegeman, 2007). In general, the messages emerging from these studies endorse the viewpoint of Hardman 
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(2002, 2005), namely that despite the various legislative efforts, school physical education is still facing some 
substantial deficiencies (e.g., unavailability and/or inaccessibility of adequate sport accommodation, discord 
between official and actual curricular time allocation) and challenges (e.g., implementation of a course-specific 
quality assurance system) that need to be addressed in order to maintain and/or upgrade the subjects’ educational 
quality. 

A similar discrepancy between policy rhetoric and actual practice appears to exist in Flanders, the northern 
Dutch speaking part of Belgium. Although school physical education is a compulsory course in this country, with 
clearly defined attainment levels and an official school inspectorate, the actual quality of the subject remains 
debatable. In fact, research of De Knop, Theeboom, Huts, Van Hoecke and De Martelaer, (2004); Huts, De Knop 
and Theeboom (2005) and Huts, De Knop, Theeboom and Van Hoecke (2007), point out that pupils, P. E. teachers 
as well as representatives from different social, educational, and workforce agencies have the opinion that the 
official attainment levels are too seldom being achieved. According to these respondents, the most important 
reasons for not attaining the predetermined levels are generally process and/or input related. This means that 
actual quality is mainly determined by factors related to the educational system and processes (e.g., lack of 
curricular time allocation, low subject status, teaching of too large classes). 

Next to this, an increased general interest in the functioning and actual realized quality of education (Devos, 
et al., 2000; Hendriks, 2001), the growing quality awareness of parents and pupils (Hendriks, 2001; Vertommen, 
2000), the need to rationalize or respond more efficiently to the ever changing expectations of contemporary 
society (Cremers-Van Wees, Rekveld, Brandsma & Bosker, 1996; Vanwing, 2000), the pursuit of lower 
educational costs (Smolders & Jegers, 1994), and the deregulation of the educational scenery (Liket, 1992; Olthof, 
1993; Vanhoof & Van Petegem, 2006) have, among others, pressured the Flemish educational government to 
change to a pedagogic-didactical approach for assuring educational quality (Devos, et al., 2000; Peeters, 2002; 
Standaert, 2001). An important characteristic of this reform is a shift in the locus of control from government 
towards school level. As a consequence, Flemish schools have become more autonomous but also more 
accountable for the quality and performance (effectiveness) of the education provided (De Medts, 2006). To come 
to terms with this increased responsibility and independence, schools have to develop and systematically 
implement an internal quality assurance policy, attuned to the principles of contemporary performance and quality 
management. In order to help schools with this task, different research projects, with regard to the identification 
and/or listing of critical success factors, have been generated and many types of school self-evaluation instruments 
have been developed to analyze different elements of the educational setting (e.g., staff, physical setting, 
organization and controlling of processes) (Cremers-van Wees, et al., 1996; Deckers & Jacobs, 1994; Devos, et al., 
2000; Gallegos, 1994; Hendriks, 2001, 2003; Voogt, 1995). 

In line with the former initiatives and conform with the total quality definition of Dahlgaard, Kristensen and 
Kanji (1995) (i.e., continuous improvement in which everyone actively participates), Van Cauwenberghe (2003) 
and Van Cauwenberghe, Vergauwen and Behets (2003, 2004) stressed that the school physical education 
department and the individual P. E. teacher also play an important role in assuring educational quality, and more 
specifically school physical education quality. To achieve this, it would be helpful to provide these P. E. teachers 
with a guiding tool that makes them aware of quality (assurance) management and convinces them to install a 
well-organized and successful subject. However, contrary to the multitude of evaluation instruments for schools 
(Berckmoes, 2007; Durieux & Ongenaert, 2005; Van Petegem & Cautreels, 2003; Van Petegem & Jacobs, 2006), 
to date there appear to be no suitable course-specific tools that can help departments and/or teachers with the 
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unavoidable process of professionalization. 

1.2 Purpose of the research project 

Considering the foregoing argumentation, the objective of the present study was twofold. Firstly, as part of a 
large research project on the quality of school physical education in Flemish secondary schools (De Knop, et al., 
2004; Huts, De Knop & Theeboom, 2005; Huts, et al., 2007), it aims at providing a preliminary picture of the 
product and process quality of physical education in a sample of Flemish secondary schools. Following the total 
quality framework of van Bottenburg, van’t Hof and Oldenboom (1997), school physical educations’ product and 
process quality can be considered as adequate when the course fulfils the criteria laid down by experts (product 
quality) and is organized effectively and efficiently (process quality). 

Next to this, for reasons of supporting future quality assurance processes in school physical education, this 
study seeks to evaluate the functionality, applicability and user-friendliness of the utilized instrument (e.g., value, 
meaning, clarity). The researchers opted to develop a multifunctional instrument that could not only be applied for 
collecting relevant research data, but that could also be used in actual day-to-day practice. For instance, at school 
level, the instrument should provide an overview of the quality determining factors (i.e., critical success factors 
with regard to the educational system and process) and be available as a frame of reference for conducting 
course-specific self-evaluations in school physical education. In this context, it should clearly discern the strengths 
as well as the shortcomings in achieving well-functioning processes and facilitate a structured remediation in 
order to optimize the overall effectiveness and efficiency of the course. Next to this, while considering the 
sometimes problematic status of the subject (De Knop, Theeboom, Huts, De Martelaer & Cloes, 2005; Hardman 
& Marshall, 2000; Zeigler, 1999), the instrument should provide P. E. departments with an extra objective 
argument to permanently confront others (e.g., school principals, government) with standards of quality school 
physical education and subsequently convince them to invest (more) in the subject. Finally, the instrument should 
allow the Flemish pedagogical counselling officers, to periodically measure physical educations’ quality progress 
in schools. 

2. Method 

2.1 Instruments 

A prototype of the course-specific (self-) evaluation instrument IKLO (i.e.. Instrument for quality analysis of 
school physical education) and a written questionnaire (see Appendix) were applied to respectively gather data on 
the process/product quality of school physical education and on the functionality/applicability of the 
IKLO-instrument. As illustrated in Figure 1, the development of the IKLO-instrument is carried out in four 
consecutive phases corresponding to the experimental design of Van Hoecke (2000). In this context, the present 
study is part of the evaluation phase (i.e., on field testing/assessment). 

Phase I: Selection of a conceptual model. The IKLO-instrument is based upon a conceptual model for quality 
and performance management in sports clubs and more particularly youth academies (see Figure 2) (Van Hoecke 
& De Knop 2006; Van Hoecke, Schoukens & De Knop, 2006). Van Hoecke (2000) reviewed and logically 
integrated various evaluation approaches (Steinmetz, 1983; Zeilhaml, Parasuraman & Berry, 1993), theoretical 
concepts (Buswell, 1998; De Meulemeester & Callewier, 1997; Lovelock, Vandermerwe & Lewis, 1999; van 
Bottenburg, van’t Hof & Oldenboom, 1997) and practical models (e.g., EFQM, CIPO, McKinsey 7S-model) in 
order to present a conceptual framework for the structured analysis of service systems and processes. 
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Figure 1 Experimental design for the development of IKLO based on Van Hoecke (2000) 
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Figure 2 The conceptual model for quality analysis of Van Hoecke (2000) 

The selected framework is based upon the (open) system and the evaluation-by-standards approach. As 
illustrated in Figure 2, it departs from the idea that all service delivery systems are dependent on input variables 
for generating processes that will ultimately result in actual output (i.e., system approach). It also suggests that 
evaluations should be based on a comparison between actual performance characteristics and predetermined 
minimal/optimal standards (i.e., discrepancy evaluation model). These standards/criteria should ultimately be 
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determined after considering different quality approaches and involving various stakeholders (i.e., 
multi-perspective approach), while judgments about actual performance should arise from the assessment of 
objective strategic and operational input variables (i.e., personnel, facilities, service design). 

Phase II: Exploration and/or analysis of relevant course-specific quality dimensions, standards and criteria. 
The selection of objective course-specific (sub-) dimensions, standards and criteria for the IKLO-instrument was 
carried out in different stages. A first draft of relevant quality dimensions, sub-dimensions, standards and criteria 
was obtained from earlier research findings (De Knop, et al., 2004; Huts, De Knop & Theeboom, 2005; Huts, et 
al., 2007; Van Hoecke, Schoukens & De Knop, 2006) together with an extensive literature and document review 
(e.g., evaluation frameworks, practical guidelines, official documents). Next, to further assure the content and 
context validity of the instrument, all the gathered (sub)dimensions, standards and criteria were delivered to 
experts (N=50) with a close and detailed knowledge of the inner workings of schools and/or the school physical 
education subject (i.e., product quality) (see Table 1). This is consistent with Chelladurai and CHANG (2000) who 
argued that, as external consumers (e.g., society, pupils) may lack the knowledge and/or expertise to fully 
understand the functioning, technical aspects and/or problems of a human service (e.g., school physical education), 
standards should primarily be conform to professional specifications. The experts were asked to carefully analyze 
the (sub)dimensions, standards and criteria, bearing in mind three questions: (1) Which are according to you 
important dimensions of quality school physical education? (2) Which are according to you important 
sub-dimensions, standards and criteria for quality school physical education? and (3) Can you make a distinction 
between minimal and/or optimal standards/criteria? 



Table 1 Demographic characteristics of the interviewees 



_ _ . Pedagogical School School „ 

P. E. teachers . . . . Experts 

, counseling inspectors principals , 

(ll = 20) officers (n= 13) (n = 2) (n = 5) ( " = 10) 



Gender 

Female 




10 




6 






1 




3 






7 


Male 




10 




7 






1 




2 






3 




Age 


Exp. 


Age 




Exp. 


Age 


Exp. 


Age 




Exp. 


Age 


Exp. 


Mean 


49.47 


26.12 


52.73 




26.50 


53.50 


24.50 


52.33 




23.33 


61.00 


25.00 


STD 


6.330 


6.547 


6.451 




7.091 


2.121 


4.950 


2.082 




4.726 


6.325 


16.971 


Max. 


59 


33 


67 




14 


55 


28 


54 




27 


69 


37 


Min. 


33 


8 


45 




41 


52 


21 


50 




18 


55 


13 



During subsequent in-depth interviews, the researchers ascertained which (sub-) dimensions, standards and 
criteria should be further adjusted (i.e., added, removed, combined and/or reformulated). The constant comparing 
and contrasting of these qualitative data (Lincoln & Guba, 1985) resulted in a total of 232 objectives, 
course-specific quality standards/criteria, related to 20 sub-dimensions and divided over six general dimensions, 
namely: 

(1) Organizational structure (10 indicators): analysis of organisational charts, departmental task descriptions 
and P. E. teachers’ commitment to attend consultative structures (e.g., advisory committee) and/or perform school 
organizational tasks (e.g., class teacher) in order to detect if the school physical education department and its 
members are firmly established in the school structure; 

(2) Strategic planning (33 indicators): analysis of strategic documents such as school mission, departmental 
vision, SWOT-analysis, long and short term objectives and financial planning; 
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(3) Facilities and equipment (81 indicators): analysis of general (e.g., availability, accessibility) and rational 
(e.g., neatness, safety) management procedures, assessment of in- and outdoor facilities and equipment; 

(4) Physical education as core product (51 indicators): analysis of the course-specific vision (e.g., content, 
methods, pupil evaluation), didactical documents (e.g., year plan, accommodation plan), possibilities for 
extra-curricular physical activity; 

(5) Internal and external communication (26 indicators): analysis of teacher/pupil feedback procedures, 
course-specific code of conduct, school magazine, info brochures, school website, collaboration plans with other 
schools, sport organisations and/or local authority; 

(6) Human resources management (31 indicators): analysis of P. E. teachers’ job descriptions, recruitment 
selection processes, evaluation and performance interviews, attendance of internal and external refresher courses, 
compensations and benefits. 

These dimensions largely correspond to the PASS model of Van Hoecke, et al (2006) for the evaluation of 
youth academies in (professional) sports clubs. 

Phase III: Operationalization of the instrument. Each quality dimension (incl. appropriate sub-dimensions, 
standards and criteria) was then organized in a separate computerized checklist. Furthermore, referring to Van 
Hoecke (2000), Van Hoecke and De Knop (2006), to make a clearer distinction between minima and optima, a 
logical weighing between the different quality dimensions, sub-dimensions, standards and criteria was proposed 
by the researchers. By conclusion, this version of the IKLO-instrument including the proposed weighing was 
revised by two representatives from each of the earlier mentioned expert groups (e.g., P. E. teachers, inspectors) 
(A=10), and discussed in a round-up meeting, until consensus about the proposed evaluation instrument (i.e., 
content and weightings) was reached. Consequently, a maximum of 2000 points, divided over the different 
dimensions (i.e., “organizational structure”: 250 points; “strategic planning”: 300 points; “facilities and 
equipment”: 300 points; “P. E. as core product”: 700 points; “internal and external communication”: 250 points 
and “human resources management”: 200 points) can be obtained in the IKLO-instrument. 

Phase IV: Evaluation of the instrument. Finally, the inter- rater reliability of the IKLO-instrument was tested 
in fifteen secondary schools using a parallel scoring method (i.e., two P. E. teachers originating from the same 
school independently completed the IKLO-instrument). Inter-rater correlations were figured out and a high 
reliability (r>0.9) coefficient and significant correlation (Kappa) (p< 0.001) were obtained for all criteria. 

2.2 Data collection 

A standardized procedure was followed to collect data in a sample of Flemish secondary schools (N= 100). 
These schools were recruited at random and stratified by educational network (e.g., catholic or community school) 
and number of pupils (i.e., n<300, 300<n<550 or 550<n). First permission to conduct the study was gained from 
the school principal and the P. E. teacher. Next, a researcher visited the school and completed the different 
IKLO-checklists based upon document analysis, school observations (e.g., available material) and a meeting with 
the P. E. teacher(s). This procedure was opted in order to increase the objectivity of the process/product quality 
analysis. 

Afterwards, a questionnaire for the functional evaluation of IKLO (see Appendix) was administered. The 
latter questionnaire could be filled in at the PE. teachers’ convenience and returned to the researchers later (degree 
of response: 82.0%). 

2.3 Data analysis 

The IKLO-data were analyzed descriptively by means of the Statistical Package for the Social Sciences 
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(SPPS 15.0). Frequency and percentage were calculated per indicator, while mean, median, minimum and 
maximum scores were figured out per quality (sub)dimension. Subsequently, two-tailed Kruskal-Wallis and 
Mann- Whitney analysis with Bonferroni adjustments were computed to estimate differences between school size, 
school location and educational network (Field, 2006). In this context, the alpha level for significance was set at 
5%. 

To analyze the data obtained with the IKLO-assessment questionnaire, descriptive statistics (i.e., frequency, 
percentage) were generated for the closed-ended questions, while inductive procedures were opted to classify the 
open-ended answers into interpretable and meaningful key themes and categories (Glaser & Straus, 1967). 

3. Results 

As reported in Table 2, the mean scores of the school audits suggest that the process and product quality of 
physical education in Flemish secondary schools can be considered as adequate. However, of all the participating 
schools, 66.0% fails on one (27.0%), two (18.0%) or even more (21.0%) quality dimensions. 



Table 2 Descriptive statistics with regard to the scores on the 6 dimensions of IKLO 



N= 100 


ORG 


STR 


FAC 


PEC 


COM 


HRM 


Mean (%) 


66.9 


52.9 


83.8 


76.5 


65.6 


55.3 


STD 


23.3 


22.7 


12.8 


18.6 


15.7 


21.6 


Maximum (%) 


100 


98.4 


99.9 


100 


100 


97.9 


Minimum (%) 


4.5 


4.0 


41.1 


16.3 


22.6 


11.2 


Median (%) 


65.3 


51.5 


88.1 


82.6 


67.1 


52.2 


Score<60.0% (%) 


43.0 


58.0 


8.0 


16.0 


33.0 


58.0 


Score<50.0% (%) 


22.0 


49.0 


1.0 


10.0 


17.0 


48.0 



Notes: ORG (organizational structure), STR (strategic planning), FAC (facilities and equipment), PEC (physical education as a 
core product), COM (internal and external communication) and HRM (human resources management). 

Generally, the lowest scores are encountered on the dimensions “strategic planning” (52.9%) and “human 
resources management” (55.3%). In fact, almost half of the research sample does not obtain a score above 50.0% 
for either of these dimensions. The low scores on “strategic planning” are mainly due to the lack of a formal 
departmental vision, the inexistence of long or short term goals, and the absence of a long and/or short term 
estimation of course-specific costs. With regard to “human resources management”, the gathered data identify the 
lack of formal job descriptions, the internal evaluation of P. E. teachers (e.g., frequency and/or nature) and the 
absence of regular departmental meetings as most common weaknesses. On the contract, the continued education 
of P. E. teachers is uncovered as an important strength of this dimension. For instance, in 91.0% of the 
participating schools information about refresher courses is carefully distributed to the P. E. teachers, and in 
8 1 .0% of the schools all P. E. teachers follow at least one external refresher course a year. In this context however, 
it is worth noting that external refresher courses are in more than half of the schools not attended in function of 
departmental or school needs, nor is information obtained during these courses automatically distributed among 
colleagues. 

The highest mean scores are registered on the dimensions “facilities and equipment” (83.8%) and “P. E. as 
core product” (76.5%). Within “facilities and equipment” high mean scores are attributed to all four 
sub-dimensions: general and rational facility management (81.5%), outdoor accommodation (94.6%), indoor 
accommodation (82.3%) and sports equipment (82.3%). A more detailed analysis of these sub-dimensions (i.e.. 
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frequencies of standards/criteria) reveals some general positive trends, such as (1) lessons that are seldom 
cancelled because of unavailable facilities (i.e., cancellation of lessons was lower than 2 hours/class/year), (2) 
lessons that, if desired, can always be taught indoor, (3) hygienic facilities that are accessible in less than five 
minutes and (4) adequate safety policies. Next to this, 91.0% of the research sample indicates that they have an 
outdoor accommodation with minimal sporting equipment at their disposal. However, the quality of the outdoor 
accommodation is considered insufficient in 31.0% of the participating schools (e.g., rough surface). On the 
contrary, most standards/criteria of the sub-dimensions indoor accommodation (e.g., dressing rooms, equipment 
store room) and sports equipment are achieved in over 70.0% of the visited schools. “P. E. as a core product” also 
scores high because in most of the participating schools: (1) there are some agreements with regard to lesson 
content (e.g., vertical and horizontal coherence), teaching methods and evaluation procedures, (2) the necessary 
didactical documents are up to standards (e.g., year plan, accommodation plan), (3) all official documents (e.g., 
final attainment levels, teaching curriculum) are available, and (4) possibilities for competitive and/or recreational 
extra-curricular physical activities are provided. 

To conclude, moderate mean scores are found on the dimensions “organizational structure” (66.9%) and 
“internal and external communication” (65.6%). Reoccurring characteristics within the dimension “organizational 
structure” are, among others, the lack of a formal departmental task description, the absence of a departmental 
organization chart and/or P. E. teachers that perform school organizational tasks (e.g., class teacher), but who are 
seldom member of a school consultative structure (e.g., advisory committee). Next to this, the moderate score on 
“internal and external communication” is generally due to a high score on internal communication (76.4%) (e.g., 
teacher/pupil feedback, course specific code of conduct) and lower scores on, respectively, external 
communication (55.3%) (e.g., presence of P. E. on open school days, on the school website or in the schools’ info 
brochure) and collaboration with other organizations (54.5%) (e.g., sport organizations, other schools). 

Kruskal-Wallis analysis suggests that four quality dimensions are affected by school size (see Table 3). 
Mann- Whitney tests were computed to follow up this finding. A Bonferroni correction was applied and so all 
differences were reported at u=0.025 level of significance. It appeared that large schools («>550) scored 
significantly higher than small (n<300) schools for the dimensions “organizational structure” ( Uorc =337.0, 
p=0.022, r=-0.28), “strategic planning” (Ustr =334.5, p=0.021, r=-0.29), “P. E. as a core product” {Upec =303.0, 
/? =0.006, r=-0.34) and “internal and external communication” ( Ucom =329.0, p =().() 1 7, r=-0.3). Large schools 
also scored higher than middle-sized schools (550<n) for the dimensions “internal and external communication” 
{Ucom= 477.0, p= 0.015, r=-0.28), and “human resources management” (JJhrm =468.0, p=0.02, r=- 0.27). 



Table 3 Descriptive statistics and Kruskal-Wallis calculations for school size 



School size 


ORG 


STR 


FAC 


PEC 


COM 


HRM 


Mean (%) 














n<300 


58.3 


46.3 


84.2 


68.9 


61.1 


50.4 


300<n<550 


65.3 


49.6 


81.6 


74.5 


62.8 


51.5 


550<n 


74.1 


60.1 


85.6 


83.4 


71.2 


61.9 


Score<50.0% (%) 














n<300 


34.6 


57.7 


0 


14.2 


26.9 


57.7 


300<n<550 


22.9 


54.3 


2.9 


11.4 


22.9 


57.1 


550<n 


12.8 


38.5 


0 


2.6 


5.1 


33.3 


Kruskal-Wallis H(2) 
P 


5.87 

0.053 


6.94 

0.031* 


4.61 

0.135 


8.79 

0.012* 


8.21 

0.016* 


6.18 

0.046* 



Note: *p < 0.05. 
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Finally, as illustrated in Table 4, Mann-Whitney analysis also revealed that, at a= 0.05 level of significance, 
catholic schools scored higher than community schools on the dimensions of “organizational structure” 
(£/ot?c=885.5, y;=0.02, /=-(). 1 1 ), “P. E. as a core product” (Upec=&10.5, yr=0.015, /--0.24), “internal and external 
communication” (Ucom =868.5, y;=0. 015, /--0.24), and “human resources management” ( L'///,>v/=853.5, p=0.0 1 1 , 
r=-0.26). 



Table 4 Descriptive statistics and Mann-Whitney calculations for educational network 



Educational Network 


ORG 


STR 


FAC 


PEC 


COM 


HRM 


Mean (%) 
Catholic 


73.3 


56.2 


86.9 


81.8 


70.1 


61.9 


Community 


62.2 


50.5 


81.6 


72.7 


62.3 


50.5 


Score<50.0% (%) 
Catholic 


9.5 


45.2 


0 


4.8 


9.5 


33.3 


Community 


31.0 


51.7 


1.7 


13.8 


22.4 


58.6 


Mann-Whitney U 


885.5 


1037.5 


957.0 


870.5 


868.5 


853.5 


P 


0.02* 


0.2 


0.68 


0.015* 


0.015* 


0.011* 


r 


-0.23 


-0.13 


-0.18 


-0.24 


-0.24 


-0.25 



Note: ’ p < 0.05. 



In general, the IKLO-instrument was labelled as a potential (46.3%) or certain (48.8%) asset for the future of 
school physical education because it can (26.8%) or will (63.4%) help improve the subjects’ quality. However, the 
respondents acknowledge that the actual benefits to be derived from IKLO are strongly dependent on the P. E. 
teachers’ motivation and discipline, the P. E. departments’ willingness to consider the instrument, and the school 
principals’ sense of responsibility. 

As illustrated in Table 5, a large majority of respondents consider IKLO to be a user-friendly, understandable 
and valid evaluation instrument. Besides these characteristics, the instrument is valued for its objectiveness, 
survey ability, clearness and completeness, which according to the respondents, makes it a stimulating and solid 
instrument for course-specific self-reflections. The immediate availability of the results (i.e., scores and charts) 
and the incorporation of a weighing, in order to facilitate a more structured remediation, are mentioned as 
additional strong points. Moreover, the respondents emphasize that the IKLO-results are to a large extent (76.8%) 
or completely (18.3%) in line with their expectations. 



Table 5 General evaluation of the IKLO-instrument (1V=82) 



Score (%) 


User-friendly 


Understandable 


Content validity 


Clear results 


Very unsatisfactory 


0 


0 


0 


0 


Unsatisfactory 


1.2 


6.1 


17.1 


2.4 


Satisfactory 


62.2 


75.6 


74.4 


72.0 


Very satisfactory 


36.6 


18.3 


8.5 


25.6 



Notwithstanding, the overall positive evaluation of IKLO, 17.0% of the respondents (see Table 5) appear to 
question the instruments’ content validity. This minority stresses that some standards/indicators are too school 
oriented (e.g., school mission) and/or not understandable (e.g., subject vision) without an additional clarification. 
They also indicate that the IKLO-instrument might provide a distorted picture because it exclusively scores 
tangible, objective measurable indicators (e.g., documents) and does not consider the many oral or informal 
agreements that are being made in practice (i.e., too bureaucratic). According to these respondents, the presence of 
documents is no guarantee for actual subject quality and is even unnecessary for some — mostly smaller — P. E. 
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departments. Other cited weaknesses of the IKLO-instrument are: (1) the inability to measure (intrinsic) teacher 
motivation, (2) the absence of a practical, output-oriented dimension (e.g., realization of the attainment levels, 
class observations), (3) the fact that only the availability and not the quality of the sport material can be scored, (4) 
the inability to evaluate different sporting facilities, and (5) the fact that results are presented in scores and not 
percentages. 

5. Discussion and conclusion 

It was the aim of this study to gain a preliminary insight into the product and process quality of physical 
education in a sample of Flemish secondary schools. With mean scores above 50.0% for all quality dimensions, 
the present research data suggest that physical educations’ process and product quality can be considered as 
adequate in the participating schools. However, this finding should be treated with caution as it might provide a 
distorted global picture of the actual day-to-day reality. In fact, scores and number of fails (i.e., scores<50.0%) 
varied considerably from one evaluated school to another. Next to this, although at random selected by 
educational network and number of pupils, all schools in the research sample participated voluntarily. Assuming 
that these schools were already more convinced about the quality of their physical education course, caution is 
required with generalizations that should be made towards the total Flemish secondary school population. In fact, 
the latter circumstance suggests that a representative score for the total population might actually be lower. 

Notwithstanding its limitations, the present study does shed a light on some possible quality tendencies in 
school physical education. On an individual level, it reveals that many of the participating P. E. teachers regularly 
attend external refresher courses, pay attention to internal communication (i.e., feedback towards pupils) and 
apply official policy documents (e.g., final attainment levels) to develop appropriate year plans. On a departmental 
level, the gathered data show that the participating P. E. teachers organize extra-curricular physical activities and 
work together with colleagues to draw up accommodation plans and/or make formal agreements regarding items 
that directly affect actual practice (e.g., coherence between subject contents, teaching methods and/or pupil 
evaluations). On the contrary, these P. E. teachers rarely distribute professional information among colleagues 
(e.g., after attending a refresher course) and/or attend a refresher course to meet departmental or school needs. 
These findings also reveal that most of the visited P. E. departments formally meet only once a trimester and 
seldom dispose of documents that underpin their professionalism and/or long term thinking (e.g., course-specific 
vision, self-evaluation report, long/short term goals, estimation of course-specific costs, departmental organization 
chart). On a school level, a mission statement that emphasizes physical activity and/or physical development, an 
official document that describes the tasks and/or responsibilities of the (P. E.) department and/or individual job 
descriptions per (P. E.) teacher are practically non-existent. On the contrary, the participating schools often obtain 
a high score on “facilities and equipment”. However, this result should be put into perspective as in the applied 
draft of IKLO: (1) facilities and accommodation are to a large extent evaluated against minimum standards, (2) 
material could merely be scored on availability and not on quality, and (3) there was no possibility to score 
different indoor accommodations. Consequently, it is assumed that the score for “facilities and equipment” is 
probably an overestimation of the actual situation. 

In all, the preceding enumeration of research findings suggests that P. E. teachers’ engagement with regard to 
the provision of quality school physical education is in essence restricted to initiatives and agreements that 
directly influence the personal class room setting. Some of the participating P. E. teachers appear to be unfamiliar 
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with quality principles and/or initiatives (e.g., strategic planning) that aim to assure and/or improve physical 
educations’ professionalism, general status and/or overall situation within the school setting. Soltani, van der Meer 
and Williams (2005) emphasize that this may be due to a lack of strong leadership, problems of convincing staff to 
take ownership of quality and/or insufficient employee engagement because of lack of knowledge and/or 
recognition of Total Quality Education principles. Especially, in the smaller schools (i.e., with one or two P. E. 
teachers) the importance and or attainability of these class exceeding initiatives appears to be questioned. On the 
contrary, the larger schools in the research sample already appear to acknowledge the need and/or added value of 
formalizing and professionalizing the school physical education subject. 

Besides a preliminary insight into the process and product quality of physical education in some Flemish 
secondary schools, the second aim of this study was to develop and implement a multi-functional, course-specific 
management tool. Based upon the SWOT-analysis in Table 6, the next paragraphs will successively focus on 
strengths and weaknesses of IKLO and reflect on opportunities and possible threats that may influence the future 
implementation of this instrument. 



Table 6 SWOT-analysis of the IKLO-instrument 


Strengths 

- innovative for school physical education 

- course-specific 

- conceptual model as frame of reference 

- measurement of tangible/formal criteria 

- valid and reliable 

- multi-functional 

- user-friendly, objective, surveyable, etc. 

- appreciated by P.E. teachers 


Weaknesses 

- limited statistical basis 

- no attention for intangible/informal criteria 

- no attention for school context 

- no output/performance related dimension 

- no guarantee for actual effectiveness/performance 

- no possibility to evaluate the quality of material 

- no possibility to evaluate different facilities 




Opportunities 

- applicable on different levels by different stake-holders 
(e.g., school principal, P. E. department) 

- introduce/stimulate internal quality management 

- systematically increase quality awareness 

- basis for a structured remediation (weighing) 

- visualization of efforts to provide quality P. E. 

- visualization of quality progress in P. E. 


Threats 

- dependence on the willingness of others 

- application for trans-school comparisons 

- reliability of self-evaluations 

- a high IKLO-score as an objective in itself 

- a high IKLO-score considered as the end-point 

- merely applied as a framework for pursuing a higher 
score on external evaluations 



Important strengths of IKLO are among others: (1) the conceptual and scientific basis used to determine and 
structure quality determining factors, (2) the measurement of tangibles and formal documents in order to obtain an 
objective and reliable result, and (3) the fact that, at present, the instrument offers a unique frame of reference for 
conducting course-specific (self-) evaluations in school physical education. Next to this, a large majority of the 
participating P. E. teachers exhibit a great interest in and appreciation for this innovative instrument (e.g., 
user-friendliness, objectivity, validity, completeness). This is important because the actual benefits and 
opportunities to be derived from IKLO are strongly dependent on the attitudes of the P. E. teachers, the P. E. 
department and the school management. When applied correctly, IKLO can stimulate and drive internal quality 
improvement processes, it can help prioritize the remediation of course-specific weaknesses and ultimately may 
contribute to a more effective and efficient realization of the predetermined objectives (e.g., final attainment levels, 
school mission). Furthermore, IKLO can be applied as an objective tool to illustrate quality related efforts and/or 
quality progress in school physical education. In this context, the instrument will hopefully contribute to the 
continuous struggle for survival, confirmation, and legitimacy of school physical education, by reducing 
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misconceptions about the subject. 

On the contrary, IKLO can also be misused, for instance, when the instrument becomes the objective instead 
of the tool for performing quality evaluations (i.e., shift in the evaluators’ focus from diagnostic self-assessment to 
the collection of scores). In practice, this may lead to: (1) schools that develop formal documents but never really 
implement them, (2) schools that consider a maximum score as the end-point for their internal quality assurance 
policies, and (3) schools that merely apply the instrument to formally prepare for external evaluations in order to 
obtain a more positive score. Therefore, the researchers like to stress that scores should only be considered as a 
visualization of problematic areas and that actual prioritization among areas of improvement should always be 
based on the needs and interests of the school, P. E. department and/or P. E. teachers. 

Finally, as illustrated in Table 6, the IKLO-instrument also has its weaknesses, as there are, the lack of a 
causal relationship between the IKLO-scores and the actual effectiveness/performance of the course, and the fact 
that possibly important intangible/informal variables are not being measured (i.e., rounding off error). However, 
based upon the experiences and comments gathered in this study, some adjustments have already been made in 
order to counter some weaknesses and further improve the applicability of IKLO. For example, in order to pay 
more attention to intangible and informal aspects, the scoring method was altered from a nominal (i.e., 
available/unavailable) to an ordinal (e.g., available in good quality/available/not available) level. By doing so, the 
researchers hope to obtain an even more accurate picture of the process and product quality of school physical 
education (i.e., mapping of oral agreements and/or quality of utilized material) and consequently provide a more 
realistic and encouraging end result. Additionally, to even further increase the comprehensibility, user-friendliness 
and validity of the instrument, clarifications with practice related examples were added to the sub-dimensions and 
some standards/criteria. To conclude, all final scores were recalculated to percentages instead of effective scores. 
A specific framework to set up a concrete action plan and a structure to separately evaluate different facilities was 
added. A manual with examples of good practices is currently being developed. 

Future research will focus on testing the sensitivity and specificity of the IKLO-instrument, for example, by 
comparing IKLO-results with those of the official school inspectorate. The inter-rater reliability of the instrument 
will further be tested by comparing IKLO-scores obtained by internal (i.e., P. E. teacher) and external (i.e., 
pedagogical counselling officer) evaluators. Next to this, the fascinating relationship between the IKLO-score and 
the actual effectiveness/outcome of the course, as well as the possibility of yet adding a seventh performance 
related dimension will be investigated. Finally, after having completed the latter research topics, the 
IKLO-instrument will be used to obtain a picture of the quality of school physical education in a broader and 
representative sample of Flemish secondary schools. 
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Appendix: 

Questionnaire (Core section) 

1. Please indicate on the scale below whether IKLO is, according to you, a user-friendly instrument? 

more than adequate adequate inadequate totally inadequate 

o o o o 

2. Please indicate on the scale below whether IKLO is, according to you, an understandable instrument? 

more than adequate adequate inadequate totally inadequate 

o o o o 

3. Please indicate on the scale below whether the courses’ professionalism can, according to you, be adequately measured with the 
IKLO-instrument? 



more than adequate 


adequate 


inadequate 


totally inadequate 


o 


o 


o 


o 



4. When prior answered ‘inadequate’ or ‘totally inadequate’, please note how, according to you, the content of the IKLO-instrument 
can be further improved? 

5. Please indicate whether IKLO can, according to you be considered as a valuable instrument for the future of school physical 
education? 

yes maybe no 

o o o 

6. Please indicate whether IKLO can, according to you contribute to quality improvements in school physical education? 

yes maybe no 

o o o 

7. When prior answered ‘no’ or ‘maybe’, please note why, according to you, this is the case? 

8. Which are , according to you, the strengths of the IKLO-instrument? 

9. Which are, according to you, the weaknesses of the IKLO-instrument? 

10. Are there, according to you, any other relevant items with regard to the IKLO-instrument that were not covered? 

1 1 . Can you judge the value of the IKLO-instrument on a score of 10? 

12. Please indicate on the scale below the, according to you, clearliness of the IKLO-results? 

more than adequate adequate inadequate totally inadequate 

o o o o 

13. Please indicate whether, according to you, the results obtained by the IKLO-instrument are as expected? 

completely more or less completely not 

o o o 

14. How are, according to you, the reactions within the P. E. department, with regard to the IKLO-results? 
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enthusiastic neutral disappointed 

o o o 

15. When prior answered ‘disappointed’, please note why? 

16. Have initiatives, to improve some of the uncovered constraints, been introduced since the audit? 

o yes, namely 

o no, because 

17. Please indicate whether these initiatives were, according to you, a direct result of the IKLO-audit? 

yes no do not now 

o o o 
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